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TO INVESTIGATE THE EFFECTIVENESS OF PROGRAMMED 
INSTRUCTION AS AN ADJUNCT TO MORE TRADITIONAL METHODS OF 
COLLEGE TEACHING* THE EXPERIMEMTORS IN THIS STUDY DEVISED 
METHODS OF TESTING RELATIVE EFFECTIVENESS OF TEACHING 
MACHINES* WORKBOOKS* SELF-PACED AND IMPOSED SCHEDULES* 
TRADITIONAL TEXTBOOKS AND SELECTED READINGS. THEY ALSO 
EXAMINED THE EFFECT OF A PRE- INSTRUCTIONAL COGNITIVE 
ORGANIZER ON SUBSEQUENT LEARNING. ALL LEARNING WAS MEASURED 
BY A 100 QUESTION MULTIPLE CHOICE TEST DEVISED FOR THE 
EXPERIMENT. AS PREDICTED* PROGRAMED INSTRUCTION PROVED 
SIGNIFICANTLY MORE EFFECTIVE THAN TRADITIONAL METHODS* THOUGH 
THE SUBJECTS DEMONSTRATED CONSIDERABLE ATTACHMENT TO THE OLD 
METHODS. THE TEACHING MACHINE DID NOT SIGNIFICANTLY IMPROVE 
SCORES OVER THE WORKBOOK AMD SELF-PACING DOES MOT IMPROVE 
MEASURABLE LEARNING OVER AM IMPOSED SCHEDULE. THE TRADITIONAL 
TEXTBOOK PRODUCED HIGHER SCORES THAN THE BOOK OF SELECTED 
READINGS OVER THE TEST GROUP AS A WHOLE. RESULTS INDICATED 
THAT A DESIRE FOR THEORETICAL KNOWLEDGE* SITTING AT THE BACK 
OF THE ROOM* CREATIVITY AND LOW LEVEL OF OPINIONATION TEWED 
TO PRODUCE HIGHER SCHORES ON THE POST-TEST. THE TABULAR 
RESULTS OF THE STUDY ARE INCLUDED. CGJ) 
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IXl Underwood (1964, 13S-315) hss suggested thmt never before in the history 

of pssrchology has there been so nuch interest in end research on huaan learning* 
Much of this nay be attributed to the developaent of prograned instruction* 

Hr 111 a— (1964, 1) declared that ..rever before has such a flurry of research 
activity accomianied the developi\.at of a new uethod of instruction* However, 
for nore than a decade, nuch of this research has been characterized by frenetic 
efforts rather than by careful thought and investigation (Stolurow, 1962)* 

Those who have been so flamboyant in describing the new instructional technology 
and who have nade such Lqrperbol^c claims for its future, have generally lacked 
sound research data aa which to base their exaggerations (DeCecco, 1964, 11)* 







lilep (1964, 205-209) has inferred that the programmed instruction movement 
has reached a plateau and that it is "undergoing a period of intellectual 
incuhation prior to some new bmeakthroughs*" 8chutz and Baker (1966, 183-185) 
felt that further progress must be preceded by careful planning, restraint, 
and resolution* They did not foresee a spectacular revolution or breakthrough* 
On the other hand. Gage and Uhruh (1967, 358-359) recently declared that 
programmed instruction is fomenting a revolution in teac h i n g and that 1* 
"threatens to render irrelevant much of the research on teaching thmt has been 
done, including much of that published since 1962*" 

Most writers agree that programs do teach (Cartier, 1963, 3-8; Gamer, 

1966, v; lorsaiight and Williams, 1963 15)* Nevertheless, many valid criticisms 

of research on programmed instruction have been made* Among these criticisms 
are the following: (1) often the programs are too brief and of poor quality, 

(2) failure to use a pretest and the use of an inadequate criterion test of 
achievement or learning, (3) the use of very small samples of subjects and 
failure to study their personal characteristics, (4) failure to control for 
notelty or Hawthorne effects, and (5) generally poorly designed studies 
lacking adequate experimental control (Carr, 1962; Gentile, 1967; Moore and 
Smith, 1962; Strong, 1964; Trow, 1963)* 

Paper presented at the 1968 Annual Meeting of the American Bducational 
fesearch ^mia^lon, Qblc^, febru^ 
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fhm ovmbelaiiic Mjority of tho prograM dovolopod aro lioMr or 
Skinnoriao CSclirsai, 19d2). 8kliuior*a operant eonditioiiiiif haa provldad tho 
thaoratical baaia for thia typo of profraaBing and for hla taachlng aacliiaaa 
(Praaaey, IMS, 2). Bovavor, aavaral vritara bara pointad out tbat thia typa 
of prograa and tba taaehing nachlnaa bava not fnlfillad Sklnnar'a proaiaaa 
(Hilgard, 1M4, 134-135; Snygg, 1M2; Stafford and Ooaba, 1M7). 

Praaaay ClM5), vho baa long naintainad that aotoinatnietion abonld ba 
uaad aa an adjunct to, or aa a aupplasant for traditional nodaa of inatmction, 
Agniii aaaartad tbat it ia thia typa of inatmcticn in Mich thara ia tha 
potential for tha braakthrou|^ in taaehing* Anong othara who hava raconnandad 
tha auxiliary uaa of prograanad inatruetion with traditional nathoda of 
inatruction ara Itaunann (IMS, 17) and Zacearia and Adana (1M4, 18(^131) • 

Baportad raaaarch baa indicatad that atudanta learn progranned natarial 
aa uall frun pmgraanart taxtbooka aa fron taaehing naebinaa (Qoldatain and 
Ootkia, 1M2, 35; Bolt and BaMoek, 1M2, 50-56; Bough and laMiB, 1M3, 
285-321). StroM (1264, 225) atatad tbat for tba far atudiaa uhich attenptad 
to control tba achaduling rariabla, tbara uaa no indication of aupariority 
for aalf-paelng aubjaeta* Trarara (1242, 223-224) and othara bara raeonnandad 
that nonoognitiua or paraonal factora ba givan nora attantion if battar 
prediction aquationa of acadenic achiauen ent ara to ba obtained* Mora 
recent concern for tba utilisation of peraonal factora includaa raporta by 
Aba (1265, 303-304), Chauncay (1265, 13-36), flyan and Morgan (12M, and 

mad Heharda (1265). Anaubal and fltxgarald (IMS, 243-242) raportad 
on atudiaa ia uhich advance eognitiua organisara facilitated learning and 
ratantion of aeaniagfui verbal natarial* The nnltipla-ehoiea, aubjaet- 
cantarad eritarion teat, givan aa a prataat, ia aaauned to ba an initial 
cognitive organJ.sar* 

Problen 

The purpoaa of thia atudy uaa to invaatigata tha diffarantial affaetivanaaa 
of prognuHMd inatruetion uhich uaa uaad aa aa adjunct to nora oonvantional 
nodaa of univaraity inatruction in a eouraa in adolaacant paychology, and to 
invaatigata tha naia affacta of aalactad cognitive, paraonal, taak, and aituar- 
tional factora on learning fron divaraa nathoda of programed inatruction* A 
aacondary purpoaa uaa to invaatigata tba prediction of lea r ni n g at the collage 
level* Vvfiff*l«g vaa inferred fron acorea nada on a 103-iten nultipla-ehoiea 
(five choicaa par iten) achia v ena n t teat, typical of final eouraa a x a nlna tiooa* 
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Hypothenes 

Stated as null. It la hypothesized that (1) there is no significant 
difference in learning from programmed material whether presented by a simple 
teaching machine or by programmed workbook; (2) students receiving an initial 
cognitive organizer do not learn significantly more from programmed material 
than those not exposed to a multi pie- choice pretest ; C3) there is no significant 
difference in learning by programmed materials for students whose schedules are 
self-paced and those students whose schedules are imposed or assigned to them; 
and (4) there is no significant difference in learning from programmed 
instruction between students assigned a traditional textbook (Ausubel, 1954) 
and those assigned a book of selected papers (Seidman, 1960). 

Procedure 

Experimental Design 

A complex 4 x 2 x 2 x 2 balanced factorial analysis of variance design 
was used to test the main effects of the independent variables of the instruc- 
tional situati<m on learning by programmed instruction. The four learning-task 
independent variables (programmed instruction presented by two types of simple 
teaching machines, by programmed workbook, and no programmed instruction for 
control subjects) and three types of learning- situational independent variables 
(type of pretest or initial cognitive organizer, type of laboratory schedule, 
anrf type of textbook) are presented in Table 1. One-half of all students were 
given the multiple-choice pretest in order to study whether it operated to 
structure students* course learnll^[ selectivity, or set to learn. The 
multiple-choice test was given as part of the final examination to all subjects 

as the criterion of learning. 

Laboratory scheduling was varied also, with one-half of all students being 
given an assigned schedule and the other one-half being permitted to select 
their schedules (self- pacing) . Likewise, one-half were assigned the Ausubel 
textbook, while the others were assigned the Seidman edited book of selected 
papers. Subjects were coded so that the sex of each student could be identified, 
as well as the identification of each student's instructor. 

Subjects and Sampling Procedure 

Originally all students enrolled in all eight sections of a semester-long 
course in adolescent psychology at The University of Texas at Austin were 
included in this study. Of the 295 who enrolled, 267 completed the course. 

available for 251 students. Of these, 170 experimental 



Complete data were 



subjects bed complete data. These experiaental subjects are the ones who 
received programed instruction and are of primary interest to this report. 
While control subjects were assigned to the same autoinstructional laboratory, 
they performed tasks other than programmed instruction. This was planned to 
control the Hawthorne effect. 

Students were rttndomly assigned to the various cells of the experimental 
design. Originally, only one teaching machine was available, so the design 
was 3 X 2 X 2 X 2, Since a second teaching machine became available early in 
the study, a small sample was taken from the original teaching machine group 
and assigned to the second machine. All combinations of the 32 experimental 
conditions were present In each of the eight sections. This was planned in 
an attempt to control the influences on achievement or learning due to differ- 
ences among the five instructors, each of whom was an experienced psychologist. 

Analysis of variance results on scholastic aptitude, reading achievement, 
and English achievement test scores, revealed that there were no initial 
significant differences in these cognitive variables among students assigned 
to the five different instructors. Also, experiemental and control subjects 
did not differ significantly on these scores, although control subjects were 
s'.'shtly favored. 

Programmed Material Used 

The programmed material, authored by Professor Pierce- Jones, ccmtained 
2348 frames in 32 chapters or lessons. It was linear or Skinnerian, eliciting 
simple, composed overt responses, and provided lms»diate knowledge of results. 

Device^ for Presenting Programs 

Mneaopticon Teaching Machine . This is a small, inexpensive, plastic 
device which uses three-by-five-inch cards on which the programs were 
presented. Generally, one frame appeared on each card. When the card was 
pulled from its storage box, the correct response for that frame was visible, 
providing immediate knowledge of results. 

Kbncept-O-Oraph Teaching Machine . This is a small plastic device with a 
knob for turning programs written on standard sheets of paper. The correct 
response for each frame was masked until the student wrote his response on a 
H 0 pi^rate sheet of paper and turned the knob to the next frame. 

Programmed Workbook . The format of the programs were identical with 
those used in the Koncept-O-Qraph Teaching Machine. Masks were provided to 
cover correct responses until the student bmd written his response to each 
frame on a separate sheet of paper. 
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Criterion Test 

The criterion test was a multiple-choice examination with 100 items, each 
with live choices. This test was developed by Professor Marsh and a deliberate 
attempt was mawe to develop a difficult test to provide adequate discrimination 
for a criterion measure of learning. The test did provide an adequate ceiling. 
Scores ranged between 23 and 76, with a mean of 50.10 and a standard deviation 
of 9.40. The SpearmanrBrown formula was used to estimate the reliability of 
the criterion test. A correlation coefficient of .70 was obtained. 

Instruments Used to Measure Personal Independent Variables 

Bdwards Personal Preference Schedule . A standardized, 225-item forced- 
choice inventory designed to assess the relative strength or importance of 
15 of Murray's needs. 

Gestalt Transf ormat ion . A twenty- item, multiple-choice test, each item 
with five choices. It is a convergent thinking test included in Guilford's 
(1959) five major groups of intellectual or cognitive abilities. 

Marsh Self-Report Social and Demographic Questionnaire . A 55-item, 
multiple-choice device, each item having between four and nine choices. It 
was developed to obtain quantitative data such as educational achievement, 
habits, attitudes, and aspirations; attitude toward programmed instruction; 
socio-economic status; background and personal information; and values. 

Nelson-Penny Reading Test . Form A, Revised Edition . A standardized 
multiple-choice reading achievement test with 100 vocabulary items and 36 
reading comprehension items. It yields separate scores for vocabulary, 
comprehension, total reading tcore, and reading rate. 

Opinionatlon Scale . A modification of Rokeach's (1960) Opinionation 
Scale. There are 40 statements to which the examinee indicates the extent 
of agreement or disagreement. 

Unusual Uses . This is a modificati<» of <Mie of Guilford's (1959) 
divergent thinking tests. Six different stimulus objects were presented 
to which each examinee was asked to list as many unusual uses for which 
each object, or parts thereof, could serve. 

Statistical Treatment of Data 

Analysis of variance was used to test the main effects of the experimental 




design. Multiple linear regression analyses were used to develop a prediction 
equation. High speed digital computers were used to perform these analyses. 
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Results, and Discussion 

Fron Table 2 it is evident that experimental subjects (those who received 
programmed instruction) achieved or learned sigpuificantly more than control 
subjects, as inferred from scores earned on the criterion test. This agrees 
with the literature cited that students do learn frc® programmed instruction. 

Anyalsis of variance results in Table 3 and Table 4 support the hypothesis 
that there is no significant difference in learning whether the programs are 
presented by teaching machine or by programmed workbook. This also supports 
cited reports of other studies investigating differential methods of presenting 
programmed material. 

The mean criterion test score of experimental subjects who received the 
same criterion test at the beginning of the course as an advance cognitive 
organizer, as indicated in Table 5, was significantly higher than the mean 
score for experimental subjects who did not receive the multipe-cholce pretest. 
Thus, the hypothesis that there is no significant difference in learning by 
programmed instruction for subjects receiving an advance cognitive organizer 
and for those subjects not exposed to this type of pretest, is rejected at the 
.05 level. 

The data in Table 6 reveal no significant differences in mean criterion 
test scores for experimental subjects whose laboratory schedule was determined 
for them, and for those who chose their own schedules for cc»pleting laboratory 
assignments. Again, this is in agreement with the literature cited, and 
supports the hypothesis that there is no significant difference in learning 
by prc^rammed instruction whether students have imposed schedules or whether 
they choose their own schedules. 

It was hypothesized that there is no significant difference in learning 
by programmed instruction between students who are assigned a traditional 
textbook (Ausubel) and those assigned a book of selected papers (Seidman). 

The data in Table 7, for experimental subjects, support this hypothesis. 
However, the data in TSble 8, for experimental and control subjects cood>ined, 
indicate that subjects assigned the traditional textbook scored significantly 
higher on the criterion test. Among possible explanations for this difference 
are: (1) Students learn more effectively from a synthesized or integrated 

textbook than from a selection of papers. However, this is not supported for 
experimental subjects only. (2) The programmed material was more similar in 
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content to that found In the traditional textbook. (3) There may be significant 
interactions which cannot be revealed by this type of statistical analysis. 

Both of the latter offer plausible explanations. 

Additional analyses of variance revealed that there were no differences 
in learning by programmed instruction due to sex, nor did students of any of 
the five participating professors achieve significantly higher mean criterion 

test scores. 

Multiple linear regression analysis of nine personal and cognitive 
variables from six different measuring devices, for experimental subjects, 
produced a prediction of achievement or learning equation which yielded an 
R of .80. In Thble 9, these variables are listed in rank order of their relative 
contribution to the prediction equation for which learning was inferred from 
scores made by experimental subjects on the criterion test. The first of 
these variables, the Nelson-Denny Reading Test Total Score, is a cognitive 
variable which would be expected to be highly relevant to learning meaningful 
verbal material. The second variable, a type of educational aspiration, 
Self-Reported Course Grade Expected, was obtained from an item in the Marsh 
Social and Demographic Questionnaire (S & D), "What grade do you expect from 
this course?” However, the contribution to the prediction equation Is spuriously 
high because this questionnaire was not developed and administered until the 
end of the semester. It is highly unlikely that responses to this item made 
at the beginning of the semester would contribute so much to the prediction 

equation. 

The third variable, Self-Reported Primary Interest in Gaining Theoretical 
or Practical Knowledge, was also obtained from an item in the S t D, ‘What I*m 
really interested in is practical knowledge that I can actually apply. Response 
choices ranged from always to never. Students indicating a relatively strong 
interest in theoretical knowledge, or a relatively low interest in gaining 
practical knowledge, tended to earn higher scores on the criterion test. The 
fourth variable, Guilford *s Gestalt Transformation Total Score, is another 
convergent thinking, or cognitive factor. The subject is preaented with a 
problem along with five choices of objects from which he is to select the best 

one for solving the problem. 

There was a negative relationship between the criterion test score and 
the fifth variable, the Need Abasement scale in the Bdwartis Personal Preference 
Schedule. Experimental subjects who manifested low needs for abasement tended 
to score higher on the criterion test. The sixth variable, Self-Reported 
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Location of Seat in Claaarooa during the Course, was obtained fr-ei responses to 
an itea in the S fc D, ''Location of your seat in this class." Contrary to 
expectations, those who sat nearer the hack of the rooa earned relatively 
higher scores on the criterion test than did those who sat nearer the front 
of the rooiB. The seventh variable, Guilford's Unusual Uses Figurative or 
Synbolic Responses, is one of the divergent thi nk i n g tests froa which creativity 
is often inferred. An exaaple of such a response for an unusual use of a key 
is, "Zt is a key to ay heart when worn around her neck." 

The eighth variable, Gelf-Reported Attitude toward Lectures and Class 
Discussi<Mis for this Course, was obtained froa an itea in the S It D, Concerning 
the lectures and class discussions in this course:," with response choices 
ranging froa "BnJoyiKl then iaaensely" to "Rather detested then." The negative 
relationship indicated, on the average, rather favorable attitudes for these 
traditional nodes of instruction. The ninth variable, Rokeach's Opinionation 
Scale Total Score, was also negatively related to criterion test scores. Since 
low scores iaply liberality, it appears that liberal students tended to achieve 
relatively better criterion test scores than did the nore opinionated 
ezperiaental subjects . 



Conclusion 

A coaplex experinental design was developed which provided for a nuodier 
of experinental treataents and controls. Randoa assignaent of students to the 
various experinental and control cells appears wO have effectively controlled 
instructor differences and Hawthorne effects. Ho sex differences in learning 
by prograaaed instruction were observed. The aain effects of the experinental 
design were tested and the results w%»re consistent with findings that have 
been generally reported by other investigators. 

Soae learner variables were obtained and analysed, both cognitive and 
noncognitive variables. Nine of these variables fomed a prediction of 
learning (as inferred froa perforaance on the criterion test) eQuation, which 
yielded an R of .80, but it was pointed out that the grade expectation 
variable was spuriously high. Ho atteapt at this tiae was aade to aake 
conclusions about the aost relevant personal or learner variables of students 
being instructed with prograaaed aaterials as an adjunct to conventional 
aethods of university instruction. Significant interactions aost likely exist 
■mon| r the sevorsl independent learner variables and these will be analysed 

thoroughly later. 

o 
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It seems very reasonable that the use of programmed Instruction as an 
adjunct to other modes of instruction holds more promise for improving 
in&tructicii than claims made about using programmed devices only. There appears 
to be no limit to the possibilities with computer adaptations. When programmed 
materials are used in addition to other instructional methods, the instructor 
can actually capitalize on the Hawthorne or novelty effect. The variety 
should increase interest and motivation, while minimizing boredom. Further- 
more, programs for more limited use would be easier and less expensive to 

develop. 



This project was supported in part by The Excellence Fund oi the 
University of Texas at Austin. 
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Tftble 1 



ScheMtic Deslfo for Ivaluating the Influences of Selected Personal, 
Task, and Situational fhctors on Learning fron Progranned 
Instruction in a Course in Adolescent Psychology. 
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Table 2 

Analysis of Variance of Criterion Test Scores for Experinental Subjects 
Beceiving Prograwied Instruction and Control Subjects 
Beceiving no Programned Instruction 



Group 


N 


Mean, Criterion 
Test Score 


SD 


F 


Experimental Subjects 


170 


51.18 


9.26 


7.06»* 


Control Subjects 


81 


47.84 


9.39 




Sun or Average 


251 


50.10 


9.41 




df s 1 and 249. 










** p < .01 level of significance. 








Table 3 







Analysis of Variance of Criterion Test Scores for Experinental 
Subjects Beceiving Programed Instruction 
by Differential Methods 



Mean, Criterion 



Group 


N 


, 

Test Score 


SO 


F 


Koncept-O-ihraph Subjects 


19 


49.47 


8.34 


0.41 


Mnenopticon Subjects 


68 


51.66 


9.39 




Prograaned Workbook 
Subjects 


83 


51.17 


9.41 





Sun or Average 170 51.18 9.26 



df s 2 and 167. 
P s n.s. 



Table 4 

Analysis of Variance ox Criterion Test Scores for Experimental 
Subjects Beceiving Frograaned Instruction by 
Teaching Machines and by Prograaned Workbook 



Group 


N 


Mean, Criterion 
Test Score 


SD 


F 


Teaching Machines Subjects 
Prograaned Workbook 


87 


51.18 


9.17 


0.0001 


Subjects 


83 


51.17 


9.41 




Suai or Average 


170 


51.18 


9.26 





df s I and 167. 
F « n.s. 



ERIC 
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Table 5 

Analysis of Variance of Criterion Test Scores for Experimental Sm lects 
with Pretest Scores and Bxperieental Subjects Who Did not 
■eceive the Critericm Test as a Pretest 



Group 


M 


Mean, Criterion 
Test Score 


SD 


F 


Experinental Subjects 
vith Pretest 


85 


52.62 


10.27 


*4.23 


Sxperiiiental Subjects, 
no Pretest 


85 


49.73 


7.93 




Sub or Average 


170 


51.18 


9.26 




df a l.aM ,168. 
«p < .05. 











Table 6 

Analysis of Variance of Criterion Test Ssores 
Sxperinental Subjects by Type of 
Laboratory Schedule Assigned 


for All 




Group 


N 


Mean, Criterion 
Test Score 


SD 


P 


Subjects vith Imposed 
Schedules 


84 


50.20 


8.71 


1.85 


Subjects with Self-Paced 
Schedules 


86 


52.13 


9.73 




Sum or Average 


170 


51.18 


9.26 




df a 1 and 168. 

F a n.s. ; p < .20. 


Table 
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Analysis of Variance of Criterion Tbsc Scores 
for All Experiaental Subjects by 
T]rpe of Textbook Assigned 







Mean, Critericai 






Group 


H 


Test Score 


SD 


F 


Subjects vith Ausubel Text 


84 


52.39 


9.47 


2.90 


Subjects vith Seidman Papers 86 


49.99 


8.95 




Sum or Average 


170 


51.18 


9.26 






df • 1 and 168. 

F a n.s. ; p < .10. 
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Table 8 



Analysis of Variance of Criterion Test Scores 
for All Subjects by Type 
of Textbook Assigned 



Group 


N 


Mean, Criterion 
Test Score 


SD 


F 


Subjects with Ausubel Test 


125 


51.74 


9.25 


7.80** 


Subjects with Seidman 
Papers 


126 


48.47 


9.33 




Sum or Average 


251 


50.10 


9.41 





df a 1 and 249. 

< .01 level of significance 



Table 9 

Mine Personal and Cognitive Vbriables in R a nk 
Order of Relative Contribution to a Prediction 
of Learning E(|uation CR » *80) for 170 
Experimental Subjects Receiving 
Programmed Instruction 



Melsott-Denny Reading Test Total Score 
Self-'Reported Grade Expected in Course 

Self-Reported Primary Interest in Gainiri? Theoretical Knowledge 
or Practical Khowledge 

Gestalt Transformation Total Score 

Edvards Personal Preference Schedule's Heed Abasement 

Self-Reported Location of Seat in Classroom during the Course 

Ghusual Uses Figurative or Symbolic Responses 

Self-Reported Attitude toward Lectures and Class Discussions 
for the Course 

Opinionation Scale Total Score 
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